Classification and encoder selection based on content

ABSTRACT

In various embodiments, methods and systems are disclosed for dynamic runtime implementation and end-to-end biased tuning of a two stage image classification system based on a decision function that uses network packet sizes and multiple image characteristics to determine the selection of an encoding codec to reduce overall network bandwidth consumption.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/753,876, filed on Jan. 30, 2013, currently pending, which is acontinuation of U.S. patent application Ser. No. 12/751,065, filed onMar. 31, 2010, now U.S. Pat. No. 8,385,666, the entire contents of whichare incorporated herein by reference.

BACKGROUND

Remote computing systems can enable users to remotely access hostedresources. Servers on the remote computing systems can execute programsand transmit signals indicative of a user interface to clients that canconnect by sending signals over a network conforming to a communicationprotocol such as the TCP/IP protocol. Each connecting client may beprovided a remote presentation session, i.e., an execution environmentthat includes a set of resources. Each client can transmit signalsindicative of user input to the server and the server can apply the userinput to the appropriate session. The clients may use remotepresentation protocols such as the Remote Desktop Protocol (RDP) toconnect to a server resource.

Remote presentation compression algorithms are employed to reduce thebandwidth of the display stream to levels that are acceptable fortransmission over local area networks, wide area networks, andlow-bandwidth networks. Such algorithms typically trade off CPU time onthe server side for a lower required bandwidth. Compression algorithmsmay work well on certain image content. However, the algorithms mayperform poorly on other types of content. For example, compressionalgorithms may work well on text but not on natural images. The outputof these compression encoders may not the final stage to the network asthere may be other layers involved such as bulk compression and remotepresentation packet encapsulation. These other layers may also effectoverall bandwidth usage. In some cases an image that encodes well withone bitmap encoder (codec) may compresses poorly with the remotepresentation bulk compressor prior to network transmission.

SUMMARY

To lower overall network bandwidth usage, an encoding method may beselected based on how the image content relates to the final bandwidthused rather than using predetermined selection criteria for text,simple/block diagrams, natural images, and the like. Rather thanproviding a classifier and target (sub) codecs for each possibleclassified output, the output may be classified based on a range usingmultiple image and compression characteristics. A decision function maythen be tuned based on the network traffic conditions.

In various embodiments, methods and systems are disclosed for thedynamic runtime implementation and end-to-end biased tuning of a twostage image classification system based on a decision function that usesnetwork packet sizes and multiple image characteristics to determine theselection of an encoding codec to reduce overall network bandwidthconsumption.

BRIEF DESCRIPTION OF THE DRAWINGS

The systems, methods, and computer readable media for altering a viewperspective within a virtual environment in accordance with thisspecification are further described with reference to the accompanyingdrawings in which:

FIGS. 1 and 2 depict an example computer system wherein aspects of thepresent disclosure can be implemented.

FIG. 3 depicts an operational environment for practicing aspects of thepresent disclosure.

FIG. 4 depicts an operational environment for practicing aspects of thepresent disclosure.

FIG. 5 illustrates a computer system including circuitry foreffectuating remote desktop services.

FIG. 6 illustrates a computer system including circuitry foreffectuating remote services.

FIG. 7 illustrates an example of dividing a captured frame.

FIG. 8 illustrates an example of selecting a codec based on datacontent.

FIG. 9 illustrates an example of processing data content.

FIG. 10 illustrates an example of processing data content.

FIG. 11 illustrates an example of an operational procedure forprocessing graphics data for transmission to a client computer.

FIG. 12 illustrates an example system for incorporating aspects of thepresent disclosure.

FIG. 13 illustrates a computer readable medium bearing computerexecutable instructions discussed with respect to FIGS. 1-12.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS Computing Environmentsin General Terms

Certain specific details are set forth in the following description andfigures to provide a thorough understanding of various embodiments ofthe disclosure. Certain well-known details often associated withcomputing and software technology are not set forth in the followingdisclosure to avoid unnecessarily obscuring the various embodiments ofthe disclosure. Further, those of ordinary skill in the relevant artwill understand that they can practice other embodiments of thedisclosure without one or more of the details described below. Finally,while various methods are described with reference to steps andsequences in the following disclosure, the description as such is forproviding a clear implementation of embodiments of the disclosure, andthe steps and sequences of steps should not be taken as required topractice this disclosure.

Embodiments may execute on one or more computers. FIGS. 1 and 2 and thefollowing discussion are intended to provide a brief general descriptionof a suitable computing environment in which the disclosure may beimplemented. One skilled in the art can appreciate that computer systems200, 300 can have some or all of the components described with respectto computer 100 of FIGS. 1 and 2.

The term circuitry used throughout the disclosure can include hardwarecomponents such as hardware interrupt controllers, hard drives, networkadaptors, graphics processors, hardware based video/audio codecs, andthe firmware/software used to operate such hardware. The term circuitrycan also include microprocessors configured to perform function(s) byfirmware or by switches set in a certain way or one or more logicalprocessors, e.g., one or more cores of a multi-core general processingunit. The logical processor(s) in this example can be configured bysoftware instructions embodying logic operable to perform function(s)that are loaded from memory, e.g., RAM, ROM, firmware, and/or virtualmemory. In example embodiments where circuitry includes a combination ofhardware and software an implementer may write source code embodyinglogic that is subsequently compiled into machine readable code that canbe executed by a logical processor. Since one skilled in the art canappreciate that the state of the art has evolved to a point where thereis little difference between hardware, software, or a combination ofhardware/software, the selection of hardware versus software toeffectuate functions is merely a design choice. Thus, since one of skillin the art can appreciate that a software process can be transformedinto an equivalent hardware structure, and a hardware structure canitself be transformed into an equivalent software process, the selectionof a hardware implementation versus a software implementation is trivialand left to an implementer.

FIG. 1 depicts an example of a computing system which is configured towith aspects of the disclosure. The computing system can include acomputer 20 or the like, including a processing unit 21, a system memory22, and a system bus 23 that couples various system components includingthe system memory to the processing unit 21. The system bus 23 may beany of several types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. The system memory includes read only memory (ROM) 24and random access memory (RAM) 25. A basic input/output system 26(BIOS), containing the basic routines that help to transfer informationbetween elements within the computer 20, such as during start up, isstored in ROM 24. The computer 20 may further include a hard disk drive27 for reading from and writing to a hard disk, not shown, a magneticdisk drive 28 for reading from or writing to a removable magnetic disk29, and an optical disk drive 30 for reading from or writing to aremovable optical disk 31 such as a CD ROM or other optical media. Insome example embodiments, computer executable instructions embodyingaspects of the disclosure may be stored in ROM 24, hard disk (notshown), RAM 25, removable magnetic disk 29, optical disk 31, and/or acache of processing unit 21. The hard disk drive 27, magnetic disk drive28, and optical disk drive 30 are connected to the system bus 23 by ahard disk drive interface 32, a magnetic disk drive interface 33, and anoptical drive interface 34, respectively. The drives and theirassociated computer readable media provide non volatile storage ofcomputer readable instructions, data structures, program modules andother data for the computer 20. Although the environment describedherein employs a hard disk, a removable magnetic disk 29 and a removableoptical disk 31, it should be appreciated by those skilled in the artthat other types of computer readable media which can store data that isaccessible by a computer, such as magnetic cassettes, flash memorycards, digital video disks, Bernoulli cartridges, random access memories(RAMs), read only memories (ROMs) and the like may also be used in theoperating environment.

A number of program modules may be stored on the hard disk, magneticdisk 29, optical disk 31, ROM 24 or RAM 25, including an operatingsystem 35, one or more application programs 36, other program modules 37and program data 38. A user may enter commands and information into thecomputer 20 through input devices such as a keyboard 40 and pointingdevice 42. Other input devices (not shown) may include a microphone,joystick, game pad, satellite disk, scanner or the like. These and otherinput devices are often connected to the processing unit 21 through aserial port interface 46 that is coupled to the system bus, but may beconnected by other interfaces, such as a parallel port, game port oruniversal serial bus (USB). A display 47 or other type of display devicecan also be connected to the system bus 23 via an interface, such as avideo adapter 48. In addition to the display 47, computers typicallyinclude other peripheral output devices (not shown), such as speakersand printers. The system of FIG. 1 also includes a host adapter 55,Small Computer System Interface (SCSI) bus 56, and an external storagedevice 62 connected to the SCSI bus 56.

The computer 20 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer49. The remote computer 49 may be another computer, a server, a router,a network PC, a peer device or other common network node, a virtualmachine, and typically can include many or all of the elements describedabove relative to the computer 20, although only a memory storage device50 has been illustrated in FIG. 1. The logical connections depicted inFIG. 1 can include a local area network (LAN) 51 and a wide area network(WAN) 52. Such networking environments are commonplace in offices,enterprise wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the computer 20 can beconnected to the LAN 51 through a network interface or adapter 53. Whenused in a WAN networking environment, the computer 20 can typicallyinclude a modem 54 or other means for establishing communications overthe wide area network 52, such as the Internet. The modem 54, which maybe internal or external, can be connected to the system bus 23 via theserial port interface 46. In a networked environment, program modulesdepicted relative to the computer 20, or portions thereof, may be storedin the remote memory storage device. It will be appreciated that thenetwork connections shown are examples and other means of establishing acommunications link between the computers may be used. Moreover, whileit is envisioned that numerous embodiments of the disclosure areparticularly well-suited for computer systems, nothing in this documentis intended to limit the disclosure to such embodiments.

Referring now to FIG. 2, another embodiment of an exemplary computingsystem 100 is depicted. Computer system 100 can include a logicalprocessor 102, e.g., an execution core. While one logical processor 102is illustrated, in other embodiments computer system 100 may havemultiple logical processors, e.g., multiple execution cores perprocessor substrate and/or multiple processor substrates that could eachhave multiple execution cores. As shown by the figure, various computerreadable storage media 110 can be interconnected by one or more systembusses which couples various system components to the logical processor102. The system buses may be any of several types of bus structuresincluding a memory bus or memory controller, a peripheral bus, and alocal bus using any of a variety of bus architectures. In exampleembodiments the computer readable storage media 110 can include forexample, random access memory (RAM) 104, storage device 106, e.g.,electromechanical hard drive, solid state hard drive, etc., firmware108, e.g., FLASH RAM or ROM, and removable storage devices 118 such as,for example, CD-ROMs, floppy disks, DVDs, FLASH drives, external storagedevices, etc. It should be appreciated by those skilled in the art thatother types of computer readable storage media can be used such asmagnetic cassettes, flash memory cards, digital video disks, andBernoulli cartridges.

The computer readable storage media provide non-volatile storage ofprocessor executable instructions 122, data structures, program modulesand other data for the computer 100. A basic input/output system (BIOS)120, containing the basic routines that help to transfer informationbetween elements within the computer system 100, such as during startup, can be stored in firmware 108. A number of programs may be stored onfirmware 108, storage device 106, RAM 104, and/or removable storagedevices 118, and executed by logical processor 102 including anoperating system and/or application programs.

Commands and information may be received by computer 100 through inputdevices 116 which can include, but are not limited to, a keyboard andpointing device. Other input devices may include a microphone, joystick,game pad, scanner or the like. These and other input devices are oftenconnected to the logical processor 102 through a serial port interfacethat is coupled to the system bus, but may be connected by otherinterfaces, such as a parallel port, game port or universal serial bus(USB). A display or other type of display device can also be connectedto the system bus via an interface, such as a video adapter which can bepart of, or connected to, a graphics processor 112. In addition to thedisplay, computers typically include other peripheral output devices(not shown), such as speakers and printers. The exemplary system of FIG.1 can also include a host adapter, Small Computer System Interface(SCSI) bus, and an external storage device connected to the SCSI bus.

Computer system 100 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer.The remote computer may be another computer, a server, a router, anetwork PC, a peer device or other common network node, and typicallycan include many or all of the elements described above relative tocomputer system 100.

When used in a LAN or WAN networking environment, computer system 100can be connected to the LAN or WAN through a network interface card 114.The NIC 114, which may be internal or external, can be connected to thesystem bus. In a networked environment, program modules depictedrelative to the computer system 100, or portions thereof, may be storedin the remote memory storage device. It will be appreciated that thenetwork connections described here are exemplary and other means ofestablishing a communications link between the computers may be used.Moreover, while it is envisioned that numerous embodiments of thepresent disclosure are particularly well-suited for computerizedsystems, nothing in this document is intended to limit the disclosure tosuch embodiments.

A remote desktop system is a computer system that maintains applicationsthat can be remotely executed by client computer systems. Input isentered at a client computer system and transferred over a network(e.g., using protocols based on the International TelecommunicationsUnion (ITU) T.120 family of protocols such as Remote Desktop Protocol(RDP)) to an application on a remote desktop server. The applicationprocesses the input as if the input were entered at the remote desktopserver. The application generates output in response to the receivedinput and the output is transferred over the network to the clientcomputer system. The client computer system presents the output data.Thus, input is received and output presented at the client computersystem, while processing actually occurs at the remote desktop server. Asession can include a shell and a user interface such as a desktop, thesubsystems that track mouse movement within the desktop, the subsystemsthat translate a mouse click on an icon into commands that effectuate aninstance of a program, etc. In another example embodiment the sessioncan include an application. In this example while an application isrendered, a desktop environment may still be generated and hidden fromthe user. It should be understood that the foregoing discussion isexemplary and that the presently disclosed subject matter may beimplemented in various client/server environments and not limited to aparticular remote desktop services product.

In most, if not all remote desktop environments, input data (entered ata client computer system) typically includes mouse and keyboard datarepresenting commands to an application and output data (generated by anapplication at the remote desktop server) typically includes video datafor display on a video output device. Many remote desktop environmentsalso include functionality that extend to transfer other types of datasuch as audio, printing commands, clipboard, and the like.

Communications channels can be used to extend the RDP protocol byallowing plug-ins to transfer data over an RDP connection. Many suchextensions exist. Features such as printer redirection, clipboardredirection, port redirection, etc., use communications channeltechnology. Thus, in addition to input and output data, there may bemany communications channels that need to transfer data. Accordingly,there may be occasional requests to transfer output data and one or morechannel requests to transfer other data contending for available networkbandwidth.

Referring now to FIGS. 3 and 4, depicted are high level block diagramsof computer systems configured to effectuate virtual machines. As shownin the figures, computer system 100 can include elements described inFIGS. 1 and 2 and components operable to effectuate virtual machines.One such component is a hypervisor 202 that may also be referred to inthe art as a virtual machine monitor. The hypervisor 202 in the depictedembodiment can be configured to control and arbitrate access to thehardware of computer system 100. Broadly stated, the hypervisor 202 cangenerate execution environments called partitions such as childpartition 1 through child partition N (where N is an integer greaterthan or equal to 1). In embodiments a child partition can be consideredthe basic unit of isolation supported by the hypervisor 202, that is,each child partition can be mapped to a set of hardware resources, e.g.,memory, devices, logical processor cycles, etc., that is under controlof the hypervisor 202 and/or the parent partition and hypervisor 202 canisolate one partition from accessing another partition's resources. Inembodiments the hypervisor 202 can be a stand-alone software product, apart of an operating system, embedded within firmware of themotherboard, specialized integrated circuits, or a combination thereof.

In the above example, computer system 100 includes a parent partition204 that can also be thought of as domain 0 in the open sourcecommunity. Parent partition 204 can be configured to provide resourcesto guest operating systems executing in child partitions 1-N by usingvirtualization service providers 228 (VSPs) that are also known asback-end drivers in the open source community. In this examplearchitecture the parent partition 204 can gate access to the underlyinghardware. The VSPs 228 can be used to multiplex the interfaces to thehardware resources by way of virtualization service clients (VSCs) thatare also known as front-end drivers in the open source community. Eachchild partition can include one or more virtual processors such asvirtual processors 230 through 232 that guest operating systems 220through 222 can manage and schedule threads to execute thereon.Generally, the virtual processors 230 through 232 are executableinstructions and associated state information that provide arepresentation of a physical processor with a specific architecture. Forexample, one virtual machine may have a virtual processor havingcharacteristics of an Intel x86 processor, whereas another virtualprocessor may have the characteristics of a PowerPC processor. Thevirtual processors in this example can be mapped to logical processorsof the computer system such that the instructions that effectuate thevirtual processors will be backed by logical processors. Thus, in theseexample embodiments, multiple virtual processors can be simultaneouslyexecuting while, for example, another logical processor is executinghypervisor instructions. Generally speaking, and as illustrated by thefigures, the combination of virtual processors, various VSCs, and memoryin a partition can be considered a virtual machine such as virtualmachine 240 or 242.

Generally, guest operating systems 220 through 222 can include anyoperating system such as, for example, operating systems fromMicrosoft®, Apple®, the open source community, etc. The guest operatingsystems can include user/kernel modes of operation and can have kernelsthat can include schedulers, memory managers, etc. A kernel mode caninclude an execution mode in a logical processor that grants access toat least privileged processor instructions. Each guest operating system220 through 222 can have associated file systems that can haveapplications stored thereon such as remote desktop servers, e-commerceservers, email servers, etc., and the guest operating systemsthemselves. The guest operating systems 220-222 can schedule threads toexecute on the virtual processors 230-232 and instances of suchapplications can be effectuated.

Referring now to FIG. 4, illustrated is an alternative architecture thatcan be used to effectuate virtual machines. FIG. 4 depicts similarcomponents to those of FIG. 3, however in this example embodiment thehypervisor 202 can include the virtualization service providers 228 anddevice drivers 224, and parent partition 204 may contain configurationutilities 236. In this architecture, hypervisor 202 can perform the sameor similar functions as the hypervisor 202 of FIG. 2. The hypervisor 202of FIG. 4 can be a stand alone software product, a part of an operatingsystem, embedded within firmware of the motherboard or a portion ofhypervisor 202 can be effectuated by specialized integrated circuits. Inthis example parent partition 204 may have instructions that can be usedto configure hypervisor 202 however hardware access requests may behandled by hypervisor 202 instead of being passed to parent partition204.

Referring now to FIG. 5, computer 100 may include circuitry configuredto provide remote desktop services to connecting clients. In an exampleembodiment, the depicted operating system 400 may execute directly onthe hardware or a guest operating system 220 or 222 may be effectuatedby a virtual machine such as VM 216 or VM 218. The underlying hardware208, 210, 234, 212, and 214 is indicated in the illustrated type ofdashed lines to identify that the hardware can be virtualized.

Remote services can be provided to at least one client such as client401 (while one client is depicted remote services can be provided tomore clients.) The example client 401 can include a computer terminalthat is effectuated by hardware configured to direct user input to aremote server session and display user interface information generatedby the session. In another embodiment, client 401 can be effectuated bya computer that includes similar elements as those of computer 100 FIG.1 b. In this embodiment, client 401 can include circuitry configured toeffect operating systems and circuitry configured to emulate thefunctionality of terminals, e.g., a remote desktop client applicationthat can be executed by one or more logical processors 102. One skilledin the art can appreciate that the circuitry configured to effectuatethe operating system can also include circuitry configured to emulate aterminal

Each connecting client can have a session (such as session 404) whichallows the client to access data and applications stored on computer100. Generally, applications and certain operating system components canbe loaded into a region of memory assigned to a session. Thus, incertain instances some OS components can be spawned N times (where Nrepresents the number of current sessions). These various OS componentscan request services from the operating system kernel 418 which can, forexample, manage memory; facilitate disk reads/writes; and configurethreads from each session to execute on the logical processor 102. Someexample subsystems that can be loaded into session space can include thesubsystems that generates desktop environments, the subsystems thattrack mouse movement within the desktop, the subsystems that translatemouse clicks on icons into commands that effectuate an instance of aprogram, etc. The processes that effectuate these services, e.g.,tracking mouse movement, are tagged with an identifier associated withthe session and are loaded into a region of memory that is allocated tothe session.

A session can be generated by a session manager 416, e.g., a process.For example, the session manager 416 can initialize and manage eachremote session by generating a session identifier for a session space;assigning memory to the session space; and generating system environmentvariables and instances of subsystem processes in memory assigned to thesession space. The session manager 416 can be invoked when a request fora remote desktop session is received by the operating system 400.

A connection request can first be handled by a transport stack 410,e.g., a remote desktop protocol (RDP) stack. The transport stack 410instructions can configure logical processor 102 to listen forconnection messages on a certain port and forward them to the sessionmanager 416. When sessions are generated the transport stack 410 caninstantiate a remote desktop protocol stack instance for each session.Stack instance 414 is an example stack instance that can be generatedfor session 404. Generally, each remote desktop protocol stack instancecan be configured to route output to an associated client and routeclient input to an environment subsystem 444 for the appropriate remotesession.

As shown by the figure, in an embodiment an application 448 (while oneis shown others can also execute) can execute and generate an array ofbits. The array can be processed by a graphics interface 446 which inturn can render bitmaps, e.g., arrays of pixel values, that can bestored in memory. As shown by the figure, a remote display subsystem 420can be instantiated which can capture rendering calls and send the callsover the network to client 401 via the stack instance 414 for thesession.

In addition to remoting graphics and audio, a plug and play redirector458 can also be instantiated in order to remote diverse devices such asprinters, mp3 players, client file systems, CD ROM drives, etc. The plugand play redirector 458 can receive information from a client sidecomponent which identifies the peripheral devices coupled to the client401. The plug and play redirector 458 can then configure the operatingsystem 400 to load redirecting device drivers for the peripheral devicesof the client 401. The redirecting device drivers can receive calls fromthe operating system 400 to access the peripherals and send the callsover the network to the client 401.

As discussed above, clients may use a protocol for providing remotepresentation services such as Remote Desktop Protocol (RDP) to connectto a resource using terminal services. When a remote desktop clientconnects to a remote desktop server via a remote desktop server gateway,the gateway may open a socket connection with the remote desktop serverand redirect client traffic on the remote presentation port or a portdedicated to remote access services. The gateway may also performcertain gateway specific exchanges with the client using a remotedesktop server gateway protocol transmitted over HTTPS.

Turning to FIG. 6, depicted is a computer system 100 including circuitryfor effectuating remote services and for incorporating aspects of thepresent disclosure. As shown by the figure, in an embodiment a computersystem 100 can include components similar to those described in FIG. 2and FIG. 5, and can effectuate a remote presentation session. In anembodiment of the present disclosure a remote presentation session caninclude aspects of a console session, e.g., a session spawned for a userusing the computer system, and a remote session. Similar to thatdescribed above, the session manager 416 can initialize and manage theremote presentation session by enabling/disabling components in order toeffectuate a remote presentation session.

One set of components that can be loaded in a remote presentationsession are the console components that enable high fidelity remoting,namely, the components that take advantage of 3D graphics and 2Dgraphics rendered by 3D hardware.

3D/2D graphics rendered by 3D hardware can be accessed using a drivermodel that includes a user mode driver 522, an API 520, a graphicskernel 524, and a kernel mode driver 530. An application 448 (or anyother process such as a user interface that generates 3D graphics) cangenerate API constructs and send them to an application programminginterface 520 (API) such as Direct3D from Microsoft®. The API 520 inturn can communicate with a user mode driver 522 which can generatesprimitives, e.g., the fundamental geometric shapes used in computergraphics represented as vertices and constants which are used asbuilding blocks for other shapes, and stores them in buffers, e.g.,pages of memory. In one embodiment the application 448 can declare howit is going to use the buffer, e.g., what type of data it is going tostore in the buffer. An application, such as a videogame, may use adynamic buffer to store primitives for an avatar and a static buffer forstoring data that will not change often such as data that represents abuilding or a forest.

Continuing with the description of the driver model, the application canfill the buffers with primitives and issue execute commands. When theapplication issues an execute command the buffer can be appended to arun list by the kernel mode driver 530 and scheduled by the graphicskernel scheduler 528. Each graphics source, e.g., application or userinterface, can have a context and its own run list. The graphics kernel524 can be configured to schedule various contexts to execute on thegraphics processing unit 112. The GPU scheduler 528 can be executed bylogical processor 102 and the scheduler 528 can issue a command to thekernel mode driver 530 to render the contents of the buffer. The stackinstance 414 can be configured to receive the command and send thecontents of the buffer over the network to the client 401 where thebuffer can be processed by the GPU of the client.

Illustrated now is an example of the operation of a virtualized GPU asused in conjunction with an application that calls for remotepresentation services. Referring to FIG. 6, in an embodiment a virtualmachine session can be generated by a computer 100. For example, asession manager 416 can be executed by a logical processor 102 and aremote session that includes certain remote components can beinitialized. In this example the spawned session can include a kernel418, a graphics kernel 524, a user mode display driver 522, and a kernelmode display driver 530. The user mode driver 522 can generate graphicsprimitives that can be stored in memory. For example, the API 520 caninclude interfaces that can be exposed to processes such as a userinterface for the operating system 400 or an application 448. Theprocess can send high level API commands such as such as Point Lists,Line Lists, Line Strips, Triangle Lists, Triangle Strips, or TriangleFans, to the API 420. The API 520 can receive these commands andtranslate them into commands for the user mode driver 522 which can thengenerate vertices and store them in one or more buffers. The GPUscheduler 528 can run and determine to render the contents of thebuffer. In this example the command to the graphics processing unit 112of the server can be captured and the content of the buffer (primitives)can be sent to client 401 via network interface card 114. In anembodiment, an API can be exposed by the session manager 416 thatcomponents can interface with in order to determine whether a virtualGPU is available.

In an embodiment a virtual machine such as virtual machine 240 of FIG. 3or 4 can be instantiated and the virtual machine can serve as a platformfor execution for the operating system 400. Guest operating system 220can embody operating system 400 in this example. A virtual machine maybe instantiated when a connection request is received over the network.For example, the parent partition 204 may include an instance of thetransport stack 410 and may be configured to receive connectionrequests. The parent partition 204 may initialize a virtual machine inresponse to a connection request along with a guest operating systemincluding the capabilities to effectuate remote sessions. The connectionrequest can then be passed to the transport stack 410 of the guestoperating system 220. In this example each remote session may beinstantiated on an operating system that is executed by its own virtualmachine.

In one embodiment a virtual machine can be instantiated and a guestoperating system 220 embodying operating system 400 can be executed.Similar to that described above, a virtual machine may be instantiatedwhen a connection request is received over the network. Remote sessionsmay be generated by an operating system. The session manager 416 can beconfigured to determine that the request is for a session that supports3D graphics rendering and the session manager 416 can load a consolesession. In addition to loading the console session the session manager416 can load a stack instance 414′ for the session and configure systemto capture primitives generated by a user mode display driver 522.

The user mode driver 522 may generate graphics primitives that can becaptured and stored in buffers accessible to the transport stack 410. Akernel mode driver 530 can append the buffers to a run list for theapplication and a GPU scheduler 528 can run and determine when to issuerender commands for the buffers. When the scheduler 528 issues a rendercommand the command can be captured by, for example, the kernel modedriver 530 and sent to the client 401 via the stack instance 414′.

The GPU scheduler 528 may execute and determine to issue an instructionto render the content of the buffer. In this example the graphicsprimitives associated with the instruction to render can be sent toclient 401 via network interface card 114.

In an embodiment, at least one kernel mode process can be executed by atleast one logical processor 112 and the at least one logical processor112 can synchronize rendering vertices stored in different buffers. Forexample, a graphics processing scheduler 528, which can operatesimilarly to an operating system scheduler, can schedule GPU operations.The GPU scheduler 528 can merge separate buffers of vertices into thecorrect execution order such that the graphics processing unit of theclient 401 executes the commands in an order that allows them to berendered correctly.

One or more threads of a process such as a videogame may map multiplebuffers and each thread may issue a draw command. Identificationinformation for the vertices, e.g., information generated per buffer,per vertex, or per batch of vertices in a buffer, can be sent to the GPUscheduler 528. The information may be stored in a table along withidentification information associated with vertices from the same, orother processes and used to synchronize rendering of the variousbuffers.

An application such as a word processing program may execute anddeclare, for example, two buffers—one for storing vertices forgenerating 3D menus and the other one storing commands for generatingletters that will populate the menus. The application may map the bufferand; issue draw commands. The GPU scheduler 528 may determine the orderfor executing the two buffers such that the menus are rendered alongwith the letters in a way that it would be pleasing to look at. Forexample, other processes may issue draw commands at the same or asubstantially similar time and if the vertices were not synchronizedvertices from different threads of different processes could be renderedasynchronously on the client 401 thereby making the final imagedisplayed seem chaotic or jumbled.

A bulk compressor 450 can be used to compress the graphics primitivesprior to sending the stream of data to the client 401. In an embodimentthe bulk compressor 450 can be a user mode (not shown) or kernel modecomponent of the stack instance 414 and can be configured to look forsimilar patterns within the stream of data that is being sent to theclient 401. In this embodiment, since the bulk compressor 450 receives astream of vertices, instead of receiving multiple API constructs, frommultiple applications, the bulk compressor 450 has a larger data set ofvertices to sift through in order to find opportunities to compress.That is, since the vertices for a plurality of processes are beingremoted, instead of diverse API calls, there is a larger chance that thebulk compressor 450 will be able to find similar patterns in a givenstream.

In an embodiment, the graphics processing unit 112 may be configured touse virtual addressing instead of physical addresses for memory. Thus,the pages of memory used as buffers can be paged to system RAM or todisk from video memory. The stack instance 414′ can be configured toobtain the virtual addresses of the buffers and send the contents fromthe virtual addresses when a render command from the graphics kernel 528is captured.

An operating system 400 may be configured, e.g., various subsystems anddrivers can be loaded to capture primitives and send them to a remotecomputer such as client 401. Similar to that described above, a sessionmanager 416 can be executed by a logical processor 102 and a sessionthat includes certain remote components can be initialized. In thisexample the spawned session can include a kernel 418, a graphics kernel524, a user mode display driver 522, and a kernel mode display driver530.

A graphics kernel may schedule GPU operations. The GPU scheduler 528 canmerge separate buffers of vertices into the correct execution order suchthat the graphics processing unit of the client 401 executes thecommands in an order that allows them to be rendered correctly.

All of these variations for implementing the above mentioned partitionsare just exemplary implementations, and nothing herein should beinterpreted as limiting the disclosure to any particular virtualizationaspect.

Classification Algorithm for Selecting Optimal Image Encoder Based onImage Content

The process of compressing, encoding and decoding graphics data asreferred to herein may generally use one or more methods and systemsdescribed in commonly assigned U.S. Pat. No. 7,460,725 entitled “SystemAnd Method For Effectively Encoding And Decoding ElectronicInformation,” hereby incorporated by reference in its entirety.

In various methods and systems disclosed herein, improvements to thetransmission of remote presentation graphics data to a client computermay be implemented to provide a more timely and rich user experience.The embodiments disclosed herein for encoding and transmitting graphicsdata may be implemented using various combinations of hardware andsoftware processes. In some embodiments, functions may be executedentirely in hardware. In other embodiments, functions may be performedentirely in software. In yet further embodiments, functions may beimplemented using a combination of hardware and software processes. Suchprocesses may further be implemented using one or more CPUs and/or oneor more specialized processors such as a graphics processing unit (GPU)or other dedicated graphics rendering devices.

In remote desktop scenarios the graphics content of a user's desktoplocated on a host computer (e.g., the server) is typically streamed toanother computer (e.g., the client). The server and the client willexchange the desktop graphics data in a well defined protocol or format.Microsoft's™ Remote Desktop Protocol (RDP) is an example of such aprotocol. The RDP protocol is a stream oriented protocol that may use astream based transport such as the Transmission Control Protocol (TCP)for exchanging data with the client. Protocols such as the TCP protocoltypically exhibit inconsistent throughput especially when the underlyingtransport is a wide area network (WAN) connection. If such a link isused for RDP traffic, such unpredictable throughput may result in anegative user experience because the desktop graphics data may bedelivered to the client in a time delayed fashion.

FIG. 7 depicts an example user screen 700 of a user's desktop dividedinto rectangular tiles. The desktop may be “tiled” in equally sizedbitmaps, which may then be represented as a frame. In this example, thedarkened tiles 710 and 720 represent tiles that have changed and are tobe sent to the client. Thus in this case the frame that is sent to theclient will contain two types of graphics elements. The tiles that arenot changed may be represented as empty rectangles. The tiles that havechanged may actually be sent as encoded bitmaps.

The methods disclosed herein are not limited to a particular type ofgraphics data such as bitmaps. The disclosed methods can apply to anytype of graphic object. For example, the graphics data may comprise adescription of entities to be drawn. Generally the lossless channel canbe used to notify the client of the type and nature of the data thatwill be transmitted and the expected result after rendering. The actualgraphics data can then be transmitted to the client on the lossychannel, and the client can compare what was actually received and theresult of the rendering actions to determine if any data was missing orif there is a discrepancy between what was expected in the received dataor as a result of performing the expected actions on the data. In oneembodiment the client can note any transactions not received or screenareas that have not been updated and notify the server of thediscrepancy. In other embodiments, the client make some intelligentdecisions such as determining that only a small portion of the screen ismissing and determining to wait longer for the missing data or determinethat the data is not needed.

Furthermore, while the following descriptions are provided in thecontext of remote presentation systems, it should be understood that thedisclosed embodiments may be implemented in any type of system in whichgraphics data is encoded and compressed for delivery over a network.

Remote presentation compression algorithms are employed to reduce thebandwidth of the display stream to levels that are acceptable fortransmission over local area networks, wide area networks, andlow-bandwidth networks. Such algorithms typically trade off CPU time onthe server side for a lower required bandwidth. Compression algorithmsmay work well on certain image content. However, the algorithms mayperform poorly on other types of content. For example, compressionalgorithms may work well on text but not on natural images. The outputof these compression encoders may not be the final stage to the networkas there may be other layers involved such as bulk compression andremote presentation packet encapsulation. These other layers may alsoeffect overall bandwidth usage. In some cases an image that encodes wellwith one bitmap encoder (codec) may compresses poorly with the remotepresentation bulk compressor prior to network transmission. In fact, theend result of these processes may in some cases be an encoded andcompressed payload that is larger than the original bitmap.

Many systems simply classify image data as one of two types such as textand other images. For example, referring to FIG. 8, a bitmap of a textimage 800 and a bitmap of a natural image 810 may be associated withcodec A (820) and codec B (830), respectively. The encoded data may becompressed by a compression function 840 before placing on the network850 for transmission. However, in many cases codec B (830) may not bethe optimal encoder for some bitmaps, the result being that thecompression process 840 ultimately produces data that is ill-suited forefficient transport over the network 850.

To lower overall network bandwidth usage, an encoding method may beselected based on how the image content relates to the final bandwidthused rather than using predetermined selection criteria for text,simple/block diagrams, natural images, and the like. Rather thanproviding a classifier and target (sub) codecs for each possibleclassified output, the output may be classified based on a range usingmultiple image and compression characteristics. A decision function maythen be tuned based on the network traffic conditions. Thus an encodermay be selected that in turn ultimately results in the best compressionratio at the end of the process and prior to transmission of the dataonto the network.

In various embodiments, methods and systems are disclosed for thedynamic runtime implementation and end-to-end biased tuning of a twostage image classification system based on a decision function that usesnetwork packet sizes and multiple image characteristics to determine theselection of an encoding codec to reduce overall network bandwidthconsumption.

Most classification systems classify image regions based on high levelcriteria such as frequency content which is then used to choose tightlycoupled internal sub-encoders for those image regions.

In one embodiment, each codec and all subsequent networking encodingstages may be considered as a single black box, thus allowing the box tooptimize network bandwidth consumption more accurately than can beperformed even with complete understanding of all available imageencoding codecs. The classification system itself does not output asingle type of classification (such as text or natural image) butinstead may output a range of values, thus allowing use with a widerange of backend encoders via a tunable decision function.

In one embodiment, a classification system may comprise the followingcomponents:

(A) Flexible Image Classifier (Runtime and Tuning)

(B) Tuning System

(C) Run Time Decision Function

(D) Backend RDP Bitmap Encoders

The flexible image classifier may use two primary types of image featureclassification. The first type may be rate change based and the secondtype may be a modified color histogram. Referring to FIG. 9, the firsttype using rate change may be calculated based on differences betweenimage pixels based on their color intensities. An image may be received900 for analysis and processing. Differences in selected parameters ofthe image may be calculated and tracked. For a given parameter, if adifference is detected, then a counter may be incremented 910. For RGBvalues, differences may be calculated row by row (contiguous memory) andfor each R, G or B value that is different than the previous pixelsvalue, a counter may be incremented. The various counters may be summedat the end of processing 920. For RGB values, the individual RGB changecounters may be summed at the end of processing, thus providing fourchange values.

Referring to FIG. 10, be a modified color histogram may be used toclassify data. An image may be received 1000 for analysis andprocessing. As the selected parameters are examined, occurrences of theparameters may be tracked. As new parameter values are detected, theoccurrence may be noted in the histogram 1010. For RGB values, as eachR, G, or B value is examined, the occurrence may be tracked in an arrayof known colors so that a histogram of all colors may be maintained. Atthe end of processing, this histogram may be examined and a count ofunique parameters may be compiled 1020. For RGB values, at the end ofprocessing the histogram may be examined and a count of unique colorsmay be compiled as well as counts for each of the R, G, and B colors.

The counts may be used for frequency analysis as well as color contentcomparisons without the need to perform color space conversions, edgedetection (via Laplacian operators) or temporal to frequency conversions(via FFTs). This may allow for runtime inline use of this classificationalgorithm without degrading system performance while maintainingflexibility of decision function methods.

In an embodiment of a tuning system, a set of known imagesrepresentative of the target system usage may be used and the images maybe tested against each available encoding codec and the flexible imageclassifier. Additionally, the output of each encoding codec may be bulkcompressed as it would appear on the network in a remote presentationnetwork stream. The tuning system may then take these values andexhaustively test each characteristic output in the flexible imageclassifier against the bitmap encoding sizes. These sizes may be scaledusing the bulk compression results. The result is a set of data for theresponse of the system based on the tuning characteristics output fromthe classifier. The most stable local minimum may be selected to obtainthe characteristic of interest and its threshold value.

The tuning method described above may allow for the tuning of thedecision function based on the final network traffic output rather thanusing only the codec encoded output sizes. Accordingly, the use ofcodecs that encode well but poorly bulk compress may be avoided,allowing for the overall traffic on the network to be increased.

In one embodiment, a run time decision function may be provided thatuses the output from the flexible image classifier and the tuning valuedetermined by the tuning system to select an optimal remote presentationbitmap encoder. The disclosed embodiments may be used with any type ofbitmap and other encoders that may be used in conjunction with theencoding and transmission of graphics data. Encoders may be addedexecuting the flexible image classifier and tuning system with the newencoders.

A link characteristics detector may incorporate some of thefunctionality described in common assigned U.S. patent application Ser.No. 12/719,669 filed on Mar. 8, 2010, hereby incorporated in itsentirety.

FIG. 11 depicts an exemplary operational procedure for transmittingremote presentation data from a computing device to a client computingdevice including operations 1100, 1102, 1104, 1106, and 1108. Referringto FIG. 11, operation 1100 begins the operational procedure andoperation 1102 illustrates receiving image data for transmission to aclient computing device. The image data may represent, for example, aportion of a user screen to be rendered on the client computing device.In an embodiment the image data may comprise bitmap data. Operation 1104illustrates classifying the image data so as to determine the type ofdata contained therein. Such a classification may involve determining arate of change of the image data. The classification may additionallyand optionally include maintaining a histogram of the colors in theimage. Operation 1106 illustrates selecting one of a plurality ofencoders based on the classification. The selection of the encoder mayalso consider the current network conditions. Operation 1108 illustratesencoding the received image data using the selected encoder.

FIG. 12 depicts an exemplary system for transmitting remote presentationdata to a client computer as described above. Referring to FIG. 12,system 1200 comprises a processor 1210 and memory 1220. Memory 1220further comprises computer instructions configured to transmit remotepresentation data to a client computer. Block 1222 illustrates receivingtuning parameters for selecting one of a plurality of graphics encodersconfigured to encode a plurality of remote presentation data types.Block 1224 illustrates receiving remote presentation data fortransmission to the client computer. Block 1226 illustrates classifyingthe received remote presentation data based on said tuning parameters.Block 1228 illustrates selecting one of the plurality of encoders basedon said classifying and link data indicative of network conditions.Block 1230 illustrates encoding the receive image data using theselected encoder.

Any of the above mentioned aspects can be implemented in methods,systems, computer readable media, or any type of manufacture. Forexample, per FIG. 13, a computer readable medium can store thereoncomputer executable instructions for transmitting graphics data. Suchmedia can comprise a first subset of instructions for receiving imagedata for transmission to a client computer 1310; a second subset ofinstructions for classifying the received image data 1312; a thirdsubset of instructions for selecting one of a plurality of encodersbased on said classifying, link data indicative of network conditions,and tuning parameters, wherein the tuning parameters are configured toidentify encoding characteristics that substantially optimizes acompression ratio resulting from a bulk compression process 1314; and afourth subset of instructions for encoding the received image data usingthe selected encoder. It will be appreciated by those skilled in the artthat additional sets of instructions can be used to capture the variousother aspects disclosed herein, and that the four presently disclosedsubsets of instructions can vary in detail per the present disclosure.

The foregoing detailed description has set forth various embodiments ofthe systems and/or processes via examples and/or operational diagrams.Insofar as such block diagrams, and/or examples contain one or morefunctions and/or operations, it will be understood by those within theart that each function and/or operation within such block diagrams, orexamples can be implemented, individually and/or collectively, by a widerange of hardware, software, firmware, or virtually any combinationthereof

It should be understood that the various techniques described herein maybe implemented in connection with hardware or software or, whereappropriate, with a combination of both. Thus, the methods and apparatusof the disclosure, or certain aspects or portions thereof, may take theform of program code (i.e., instructions) embodied in tangible media,such as floppy diskettes, CD-ROMs, hard drives, or any othermachine-readable storage medium wherein, when the program code is loadedinto and executed by a machine, such as a computer, the machine becomesan apparatus for practicing the disclosure. In the case of program codeexecution on programmable computers, the computing device generallyincludes a processor, a storage medium readable by the processor(including volatile and non-volatile memory and/or storage elements), atleast one input device, and at least one output device. One or moreprograms that may implement or utilize the processes described inconnection with the disclosure, e.g., through the use of an applicationprogramming interface (API), reusable controls, or the like. Suchprograms are preferably implemented in a high level procedural or objectoriented programming language to communicate with a computer system.However, the program(s) can be implemented in assembly or machinelanguage, if desired. In any case, the language may be a compiled orinterpreted language, and combined with hardware implementations.

While the invention has been particularly shown and described withreference to a preferred embodiment thereof, it will be understood bythose skilled in the art that various changes in form and detail may bemade without departing from the scope of the present invention as setforth in the following claims. Furthermore, although elements of theinvention may be described or claimed in the singular, the plural iscontemplated unless limitation to the singular is explicitly stated.

1. A computer-implemented method for classifying image data fortransmission over a computer network, the method comprising: receiving aplurality of images representative of a target system's graphics usage;testing the plurality of images using a plurality of encoding codecs;encoding the plurality of images using the plurality of encoding codecs;compressing the encoded plurality of images to generate encoded data fortransmission over a network stream; and testing the compressed encodedplurality of images to determine a set of characteristic responses. 2.The computer-implemented method of claim 21, wherein said imageclassifier is configured to classify image data based on the input.3-20. (canceled)
 21. The computer-implemented method of claim 1, furthercomprising selecting a subset of the characteristic responses for inputto an image classifier.
 22. A computer-implemented method fortransmitting graphics data from a first computing device to a secondcomputing device, the method comprising: classifying, by the firstcomputing device, image data for transmission to the second computingdevice based on parameters for selection of one of a plurality ofgraphics encoders, the parameters derived based at least in part onanalysis of image data; selecting, by the first computing device, one ofthe plurality of graphics encoders based at least in part on saidclassifying and data indicative of network conditions; and encoding, bythe first computing device, the image data using the selected graphicsencoder.
 23. The computer-implemented method of claim 22, wherein saidselected encoder substantially optimizes a compression ratio resultingfrom bulk compressing the encoded image data.
 24. Thecomputer-implemented method of claim 22, wherein said classifyingcomprises determining a rate of change of the image data.
 25. Thecomputer-implemented method of claim 24, wherein said determiningcomprises determining a rate of change of RGB values.
 26. Thecomputer-implemented method of claim 22, wherein the image data isbitmap data.
 27. The computer-implemented method of claim 22, whereinthe method is performed in conjunction with a remote presentationprotocol.
 28. A computing device comprising at least one processor andat least one memory communicatively coupled to said at least oneprocessor when the computing device is operational, the memory havingstored therein computer-executable instructions that, upon execution bythe processor, cause: receiving encoded graphics data from a server overa communications network; and decoding the encoded graphics data,wherein the encoded graphics data is encoded using a graphics encoderselected based on classification of the graphics data and link dataindicative of network conditions of the communications network, whereinthe classification is performed based on parameters for selecting one ofa plurality of graphics encoders, the parameters derived from analysisof image data.
 29. The computing device of claim 28, wherein saidclassification comprises determination of a rate of change of thegraphics data.
 30. The computing device of claim 28, wherein saidclassification comprises maintaining a color histogram.
 31. Thecomputing device of claim 28, wherein the selected graphics encodersubstantially optimizes a compression ratio resulting from bulkcompressing the encoded graphics data.
 32. The computing device of claim28, wherein the plurality of graphics encoders are substantiallyoptimized for one or more types of image data.
 33. The computing deviceof claim 29, wherein said determination comprises determining a rate ofchange of RGB values.
 34. The computing device of claim 33, wherein saiddetermining comprises calculating differences by pixel rows and for eachR, G or B value that is different than a previous pixel value,incrementing a counter.
 35. The computing device of claim 34, furthercomprising summing individual RGB change counters.
 36. The computingdevice of claim 30, wherein said maintaining comprises maintainingoccurrences of color values in an array of colors.
 37. The computingdevice of claim 36, further comprising determining a count of uniquecolors and counts for each of R, G, and B color.
 38. The computingdevice of claim 37, further comprising using the counts for frequencyanalysis and color content comparison.